
10 A ^METHOD TO CONVERT TWO DIMENSIONAL MOTION PICTURES 



FOR THREE-DIMENSIONAL SYSTEMS 
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20 _i * - . TECHNICAL FIELD 



J^lhe invention relates to a method for converting existing film or 
videotape motion pictures to a form that can be used with three£^ 
dimensional systems for broadcast or exhibition. 

25 



BACKGROUND ART 

30 j$/ith the advent of stereophonic sound, various techniques were 
developed to convert or 're-process 1 existing monophonic programs 
for stereophonic bro^dpas t^oj^ recording systems. These included 



modifying the gMfinTTzataorfc, phase or tonal qualities o fs ^S-rot- e 
copies of the monophonic program for the left and right channels. 
35 While true stereophonic or binaural effects may not have been 
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achieved, the effects were much improved over feeding the 
identical monophonic signal to both channels. 

Similarly, with the almost universal use of color production, 
exhibition and broadcast systems for motion /ffr rn r and 
television, systems have been developed to convert exi8ting 



monochrome or b 1 ack (Ajwhite. materia 1 8 to color programs. Such a 
system^is described in applicant Geshwind^s patent number 
^4,606,625, issued August 19, 1986. The results of these colorized 



10 products, while not always identical to true color motion 
pictures, are more suitable than black (ijwhite for color systems. 



There have been a number of systems for exhibition or display of 
left- and right-eye pairs of binocular motion pictures. Early 
15 systems reqired two completely redundant projection or display 
systems; e.g. two film projectors or CRT television displays, 
each routed to one eye via mirrors. Other systems require either 
complicated and expensive projection or display ^^s^^or 
^V^^ expensive 'glasses' to deliver two septate images. For example: 
20 i * 

PI red- and green-tinted monochrome images are both projected 
or displayed to be viewed through glasses with left and 
right lenses tinted either red or green; 

25 (p| two full-color images are projected through mutually 
perpendicular and viewed through glasses 

with lenses that are also polarized in the same manner; 



left and right images are displayed on alternate odd and 
30 even fields (or frames) of a standard (or high scan rate) 

television CRT and are viewed through 'glasses' with 
shutters (either rotating blades or flickering LCDs, for 
example) that alternate the view of left and right eyes in 
synchrony with the odd or even fields of the CRT. 

35 
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1^ Of the above systems, the second is not at all usable with 
standard home television" receivers, the third requires very 
expensive ^'glasses^ and may f 1 ic ^ £ with standard home receivers, 
and the first produces on 1 y^G%tt?t££% z lgjt ed monochrome images. 
Further, none of the systems may be broadcast over standard 
television for unimpeded viewing without special glasses. 

Thus, until now, .e.. viewable as two-dimensional* 

^without glasses) home reception of 3-D images was not possible. 
10 However, a new system, which takes advantage of differential 
processing of left- and right-eye images in the human perceptual 
system, delivers a composite image on a standard home television 
at can be viewed as a normal 2-D picture without 
glasses. Very inexpensive glasses, with one light and one dark 
15 lens, accentuate the differential processing of the image, as 
viewed by each eye, to produce a 3-D depth effect. 

As* Practical, inexpensive;, c^mp i a ttrb 1 o (with standard TV) 3-D 
television may now become widespread. In addition to materials 
20 £ specifically produced for the new system (or other 3-D systems) 
5 conversion of standard 2-D programs to 3-D format woj^l d x>ro v ide 
additional product to broadcast using the new &e mpa t ob " Ie system 
& (or for other 3-D projection systems). 

25 

Clr SUMMARY OF THE INVENTION 

Applicant Handal^s previous application, number 479,679, filed 
30 ^March 28, 1983 and now abandoned, relates a procea^and apparatus 
for deriving left- and righ-t-eye pairs of binocular images, from 
certain types of two dimensional film materials. In particular, 
the materials must consist of • ^^^^^ J^^rrS'g round and background 
elements, such as cartoon, animation cells and background art. By 
35 taking into account the parallax between scenes as viewed by the 
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left and right eyes, two images are prepared where foreground 
elements are shifted with respect to background elements by an 
amount that indicates their depth in the third dimension. Twq^) 
dimensionsl motion pictures that consist of a series of single 
composite images could not be converted to t hree-dime^on^ 1 
format, by this technique, without first being o d > into 

various background and foreground elements. 

Once committed to two-dimensional motion pictures, the separation . 
and depth information for various scene elements, in the third 
dimension, are lost. Thus, t h e ^^^^^txnt / o? U two-diinen8ional 
image sequences into individual image elements and the generation 
of three-dimensional depth information for each such image 
element are not simple or trivial tasks, and are the further 
subject of the instant invention. 



In accordance with the invention, stsndard two-dimensional motion 
picture film or videotape may be converted or processed, for use 
with three-dimensional exhibition or transmission systems, so as 
20 to exhibLt>at least some three-dimensional or depth character- 
sties. ? ^B^^Uu r^of a single 2-D image stream into diverse 
elements is accomplished by a computer assisted, human operated 
system. (If working from discrete 2-D f i fm tnh n r ^^^^^ * such 
as animation elements, the^^^^tt^^^T' ma Y be cwHW-terh) Depth 
information is assigned to various elements by a combination of 
human decisions and/or computer analyses and resulting images of 
three-dimensional format are produced under computer control. 



30 

P£ BRIEF DESCRIPTION OF DRAWINGS 

P A method for carrying out the invention is described in the 
accompanying drawings in which: 

35 
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f Tgur e j Ki s a diagram illustrating the relationship of -a 2-D 
source film jm* ge, left and right image pairs, and a 
composite 3-D M.mage frame. 



s a ^^^^^^ c dia gram of a system for carrying out 
"a n^Mp^ijnft n t a t i on of the present invention. 




Figure 2 



10 D£r tL^ DETAILED DESCRIPTION 

P An immense amount of standard, 2-D motion y^X^^^lirffterial exists 
in the form of film and videotape. In addition, certain materials 
Sexist in the form of discrete 2-D image fuf . c o mpo^^^nt^ ^o^ h as 
15 ^animated cell and background paintings. As the use of 3-D 
^ — exhibition and broadcast systems becomes more widespread, the 
J conversion of existing 2-D programs to a format that will exhibit 
jfc at least some 3-D or depth effects, when used with 3-D systems, 
is desired. 

20 

$ Extracting individual image elements or 3-D depth information 
J$ from a ^-D film frame, or synthesizing 3-D, information for those 
elements, entirely by computer a ^trip^^ w ith artificial 
intelligence, is not now practical. Therefore, the embodiment of 
25 the invention as described herein employs a high degree of human 
interaction with the computer. However, as artificial 
intelligence progresses, a predominantly or completely automated 
system may become practical and is within the intended scope of 
the invention. 

30 

Figure 1 shows a frame 20 of a standard 2-D film or video motion 
picture, consisting of a cross-hatched background plane 21, a 
large white circle 22 in the mid-ground, and a small black square 
23 in the foreground. It is a 2-D representation of original 3-D 
35 scene 10 comprising elements 11, 12 and 13, which is Wot 
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fiavailable for direct 3-D photography. After human identification 
of individual image elements and depth assignment, the computer 
generates, from frame 10, a pair of coordinated left 30 and right 
40 images with backgrounds 31 and 41, circles 32 and 42, and 
squares 33 and 43 respectively. Note that the relative positions 
of the square and circle are different in the left and right 
images; this situation is similar to the parallax that might 
result between left- and right-eye views if one were to have 
viewed the original scene 10 directly.. Three-dimensional format 
frame 50 is generated by encoding the information in the left 30 
and riqht 40 image pair, in a manner consistent with any one of a 
^number of existing 3-D systems. The specific operation of these 
^various 3-D systems is not the subject of the instant invention. 

S Alternately, the steps of generating end encoding 3-D information 
may be combined such that 3-D format frame 50 may be processed 
directly from -2^0 frsme 20^without generating left 30 and right 
40 image pairs. In either case, g-D format frsme 50 when viewed 
by human 60 through j^-D glasses 70 is perceived as 3-D scene 80, 
containing elements 81, 82 and 83, which hss at leas'? some of the 
3-D characteristics of original 3-D scene 10. 

Various systems for the encoding, display, projection, recording, 
transmission or. viewing of 3-D images exist, and neJf^^Sfe^af" 
be developed. Specifically,' various techniques for specifying, 
encoding and viewing ^UD information may now, or come to^ exist, 
which do not make use of parallax offset and/or left and right 
image pairs and/or viewing .glasses, or w^ich embody new 
techniques or changes and " i^ i prffv m^^ ^^to^cuT rent systems. 
Further, such systems may integrate information from more than 
006 80urce frame 20 into a single resultant 3-D frame 50. The 

specifics of operation of such systems is not tfTe subject of the 
instant invention, however, preparation of Z-D program material 
for such systems is. 
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The offsets shown for elements 31, 32 and 33 in left frame 30 and 
elements 41, 42 and 43 in right frame 40 are meant to be 
illustrative and do not necessarily follow the correct rules for 
image parallax. In fact, depending upon where viewer attention is 
meant to be centered, different rules may apply. For example, one 
technique is to give no parallax offset to far background 
elements and to give progressively more parallax offset to 
objects as they get closer. Alternately, attention may be 
centered in the mid-ground with no parallax offset to mid-range 
objects, some parallax offset to close-range objects and reverse 
parallax offset to far-range objects. The particular placement of 
objects and attention point in the 3-D scene is as much an art as 
science and is critical to the enjoyment of 3-D programs and, 
in any event, is not meant to be the subject of this invention. 

Figure 2 shows a schematic of a system to implement the instant 
invention. A 2-D film or video image 10 is input to a video 
monitor 20 and to the scanning portion 41 of frame buffer 40. 
Video monitor 20 is capable of displaying either the 2-D image 
being input 10 or the output from display portion 42 of frame 
buffer 40. 

Frame buffer 40 consists of an image scanner 41 which can convert 
the input image. into digital form to be stored in a portion'of 
frame buffer memory section 44, a display section 42 which 
creates a video image from the contents of a portion of memory 
section 44, and a computer interface section 43 which allows the 
computer CPU 50 to read from and write to the memory section 44. 

Graphic input tablet 30 and stylus 35 allow the human operator to 
input position information to the computer 50 which can indicate 
the outline of individual image elements, choice of a specific 
image element or part of an image element, depth specification, 
choice of one of a number of functions offered on a 'menu 1 , or 
other information. An image cursor can be displayed on monitor 20 
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by frame buffer 40 to visually indicate the location or status of 
the input from the tablet 30 and pen 35. Text and numeric 
information can also be input by the operator on keyboard 55. 

m4r»eA, with 



Computer CPU 50 is -e-q /Z i ipfe fl. with software which allows it to 
interpret the commands and input from the human operator, and to 
process the digitized 2-D information input from 2-D frame 10 
into digital ^-D frame information, based on said human commands 
and input. Said digital 3-D frame information i8 then output by 
output interface 60 (which may be similar to frame buffer 40 or 
of some other design) to a videotape recorder 70 or to a film 
recorder 75, capable of recording 3-D format frames. 

The system as described above operates as follows. A frame of the 
2-D program is displayed on the video monitor for viewing by the 
human operator. With the tablet and stylus the operator outlines 
various 2-D image areas to indicate to the computer the 
boundaries of various injage elements to be oonora - to ly prq/cessed. 



(For materials, such as animation CTmponatrfe - O ', A at already exist 
n q SlTpffrn r, p/pTp m ft n \ n , the previous stage of the process may be 
skipped.) Depth position ■ information, in the third dimension, is 
determined for each image element, by a combination of operator 
input and computer analysis. Left and right image pairs or a ^~D 
composite image is processed by the computer, from the input 
image, based on the computer software and operator instructions. 
Depending upon the particular 3-D system to be used, left- and 
right-image, pairs may or may not be the final stage or an 




stage or bypassed entirely. Further, for some 1-D 



systems, information from more than one 2-D source frame may be 
combined into one 3-D frame (or frame pair). In any event, the 
final 3-D information is then collected on a videotape or film 
recorded. The process is then repeated for additional ^-D frames. 

Each image element may be given a uniform depth designation which 
may cause the perception of 'cardboard cut-out* characters. 
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1 Alternately, different portions of a single image element may be 
J given different depth designations with the computer interpo- 

lating depth coordinates over the entire element. For example^ an 
image element positioned diagonally in a frame may have its right 
5 edge designated to be closer than its left edge. Alternately, one 
feature of an image element, say a person's nose in a close-up, 

3 

might be designated as being closer than the rest of the i^age 
element. In such manner, the computer would be instructed to 
interpolate and process depth information over the entire image 
10 element. Such processing of the image element, in accordance with 
varying depth information, may result in the stretching, skewing 
or other distortion of the image element. 

Depth interpolation may also be carried out over time, between 
15 frames. For example, an image element might be designated tq be 
close in one frame and to be far away in a later frame. The d$pth 
of the image element may be interpolated for the frames between 
the two designated frames. Further, the position of the im&ge 
element in the two dimensions of the film frame, and the shap§ of 



20 the outline, cifpffr a 1 1 n g. t rye image element from the rest of the 
source image, may also be interpolated over time, between frames. 
Linear pnd non-linear interpolation techniques are well known and 
may be readily applied. 



25 It is also possible to add random noise to the depth information 
to eliminate the appearance of flat objects moving in space and 
to help achieve greater realism. 

For a particular image element, depth position may be derived by 
30 the computer, alone or with operator input, by measuring the size 
of the image element in a frame. For example, once a person were 
outlined, his overall height in the ^-D film frame might be 
extracted by the computer as an indication of depth distance. 
Using knowledge, or making assumptions, about the object's rejal 
35 size, and the characteristics of the camera and lens, the 
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and assigned depth 



distance of the object from the camera may be calculated by 
applying well known principles of perspective and optics. 
Alternately, if overall size cannot be easily extracted by the 
computer, key points might be indicated by the operator for the 
computer to measure. Comparing the change of size of an image 
element in several frames will provide information about the 
movement of the object in the third dimension. 

It should be noted that horizontal parallax offsets are, by far, 
more obvious, due to the fact that our eyes are t 6^?^i#^f ;/ the 
horizontal direction. However, vertical offset differences 
between left- and right-eye views may also be appropriate. 

As various image elements are 
values, a situation develops where diverse objects exist in a 
'three-dimensional space" within the computer. It should be noted 
that, in order J^ i8 P la Xa rea V,stic representation of the 
entire scene, Mi^T^i&k 0 o^e < i?B^u e t obscure all or part of 
rearmost objects *fith whi^ they overlap (except in the case 
where a H&H&d&i&lo&ject were transparent). When generating 
left- and right-eye views, the pattern of overlap of image 
elemen.ts, and thus the pattern of obscuring of image elements 
will, in general, be different. Further, for image elements that 
have been assigned non-uniform depth values (e.g., image elements 
that are not flat or not in a plane perpendicular to the third 
dimension) the intersection between these image elements, for the 
purpose of one obscuring the other, may be non-trivial to 
calculate. However, there .are well known techniques, that have 
been developed for computer image synthesis, that allow for the 
sorting, intersection and display of diverse, overlapping 3-D 
image elements. -0 



As image elements are^ ^offirotod . from background scenes or each 
other, holes may develop. The computer software will compensate 
35 for this by filling in missing information in one particular 2-D 
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1 frame with the equivalent part of the image element from earlier 
or later frames whenever possible. Alternately, material to 'plug 
holes^ may be created by the operator or by the computer 'from 
adjacent areas or may be newly synthesized. Further, as two ijpage 
^ 5 elements are seperated from each other, the ^mtw^^^f^t^e two 
may be moved closer to the viewer; hence, it may be appropriate 
to make it larger. Thus, the increased size may cover the gaps in 
the rearmost element of the two. 

10 As part of the processing to be performed on the 2-D source 
image, individual image elements or composite images, additional 
effects may be programmed into the computer to heighten the sense 
of depth. For example, shadows cast by one image element on 
another element or the background may be calculated and inserted; 

15 far distant landscape elements may be made hazy or bluish to 
indicate remoteness; different image elements may be blurred or 
sharpened to simulate d ep t h- o f - f i el d ; *o r , the distance between 
close objects may be fe^^^^^^^Phas iz e thet^^v^j^ 
There are, of course, other technical or artistic techniques that 

20 can be used to indicate depth in an image and which may also be 
incorporated into the image processing programs and would 
there fp re be part of the invention as described herein. 
Therefore, the above examples are illustrative and should not be 
construed as limiting the scope of the invention. Alternately, 

25 depth information may be intentionally distorted foreffect or 
for artistic purposes. 

fa-^" l -m pro rvm en - t -o.* may be made ta the final product by including new 
image elements that were not part of the original Z-D source 
30 image. These could include Z-D image elements that are then 
assigned depth values, 3^D image elements created by Jg-D 
photography and then entered into the computer as left- and 
right-image pairs, for example, or synthetic 3-D computer 
generated graphics. In particular, since computer g^erated image 
35 elements can be created with depth information, they can be 
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easily integrated , into the overall 1-D scene with vivid effect. 
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For example, a 3-D laser blast could be cheated by computer image 
synthesis suchHhat it would in turn (Sb-e-trr-e and be obscured by 
other image elements in an appropria te^^^^^^^iV^might even be 
5 created so aa to appear to continue beyond the front of the 
screen into 'viewer space 1 . 

a/ Animated film cremptrnont o usually ha vereLati vely few background 



paintings, which are kept *~fi& ^%ffi& VfYoib the animated characters 
10 £in the foreground. For these, or for live ^"DXLl^ed scenes, once 
the foreground elements have been ^^^l^r^ithe background, 
g the flat 2-D backgrounds may be replaced by 3-D backgrounds. The 
3-D backgrounds might consist of computer generated graphics, in 
which case depth information for the various elements of the 
15 background would be available at the time of the background 
^creation. Alternately, 3-D backgrounds might be created using 3-D 
photography, in which case depth information for the background 
elements may be derived, by the computer, from the comparison of 
$ the left- and right-image pairs of the 3-D background photographs 
20 and applying known image processing and pattern recognition 
^techniques. Alternately, depth information to create 3-D 
backgrounds may be specified otherwise by operator input and/or 
computer processing. Once depth information is available for the 
various background elements, it may compared to depth information 
25 for the other image elements with which it is to be combined in 
^ each frame. Thus, the 3-D background may itself have some depth 
and, in effect, be a 'set 1 within which other image elements may 
be positioned in front qf, behind or intersecting various 
background elements for added realism. 

30 

It should be noted that, intended purpose and software algorithms 
& aside, the design and operation of the 3-D conversion system 
described herein has many similarities with the black \&Ji white 
footage colorization system described in applicant Geshwin^s 
35 ^patent 4,606,625. They are both computer-aided systems which 
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1 display standard film i/^es to /8 n operator, allow the operator, 
to input information ^p^M^^-fTloua image elements within the 
V frame, allow the operator to specify attributes (color in one 

case, depth in the other) for the image elements, and cause the 
5 computer to process new image frames from the original, based on 
£> the operator input. Thus, the processing of 2-D black and white 
footage to add both color and depth information wouldji* much 
more efficient than implementing each procese^So^^fyT^y 

10 The scope of the instant invention is the conversion or 
processing of 2-D program material for use with any 3-D 
exhibition or distribution system now in use or later developed, 
^but not the specific method of operation of any particular 3-D 
system. 
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It will thus be seen that the objects set forth above, among 
those made apparent from the proceeding description, are. 
efficiently attained and, certain changes may be made in carrying 
out the above method and in the construction set forth. 
Accordingly, it is intended that all matter contained in the 
above description or shown in the accompanying figures shall be 
interpreted as illustrative and not in a limiting sense. 

C/rf Now that the invention has been described, what is claimed as new 
25 and desired to be secured by Letters Patent is: 
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